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Do Improved Examination Scores Indicate Acquired Skills? 

- A Study in Teaching the Use of Articles in Journal Writing 
Through a Continuous Testing Regimen — 

Tadashi SHIOZAWA* 

Thomas SIMMONS** 



I . Introduction 

For tin* last b years llu* authors have been iuvosliKating master v iearnniK as a 
Koal in our teaehiiiK to establish some gains that the students can take with them when 
they leave schools (Simmons, 1992; Shiozawa 1992). We have observed that discrete 
variables are comparatively easier to teach in the test retest approach (Dempster, 1992, 
pp. 212 217) because they are retained better by the students. Additionally, discrete 
variables are easier to manage because they can be dealt with in short, focused cloze 
tests or multit)le choice tests that can lie discussed ciuickly when the students have 
finished. However, while we have been able to verify the efficacy of the test retest 
approach in inipioving the students’ scores and its usefulness in managing the program- 
me's heavy schedule and large class loads, the ctuestion remains as to whether the stu- 
dents are effectively incorporating the tested objectives into their writing assignments. 
An initial study of the student's journals written during the previous year was con- 
ducted after they had been tested and retested for the use of deictic variables in writing 
but had not beem given any specific directions t(^ incorporate what they were learning 
into their journals which were being written at the same time. The preliminary survey 
indicated that they were incorporating the objectives that were being taught, but that 
they still had some ways to go before they had mastered the use of the objectives. It was 
the purpose of this study to t‘stablish wluTher or not the students can be given an 
advantage in U'arning and employing the objectives by examining the possible influence 
that using the continuous t(.‘sting has on their use of the test('d variables in their writing. 
In oth(‘r words, we set out to learn whether or not the students incorporate the discrete 
variables taught through a eontinuous testing approach into their writing assignments. 
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I . Institutional parameters and the rationale for this study 

One of the difficulties faced in conducting this research was that tlie syllabus 



objectives that were part of the mandatory learnin.u objectives. Administrative require 
nients aside, the proper use of these discrete objectives is also considered important in 
oral and written communication since they are noticeable when used incorrectly and 
tend to be annoying and frequently the cause of ambiguity. But they are tedious to study 
at length and the students are often distracted if the objectives are given more than 10 
to If) minutes during class: an attitude that is a further detriment to the classrooni en 
vironment. Reducing the time spent on such objectives could possibly decrease the stu- 
dents' lack of enthusiasm for learning them and arguably boost teacher morale. 

.Additionally, in the course of teaching large classes it becomes necessary to deal 
efficiently and quickly with an overwhelming number of classroom management vari- 
ables. Thus, it is expedient to deal with discrete objectives quickly and consistently so 
that the students will have an opportunity to master them and the teacher will be able to 
handle them \n the inadecpiate time allowed for an accumulating number of objectives 
through the course of the year. So. it is important to find straightforward ways of deal 
ing with discrete objectives in a well delineated format that can be taught with a mini- 
mum of preparation using a teachers* resource bank, evaluated quickly and produce con 
sistent results. Such an anaiigement will significantly reduce the classroom teachers' 
workload, thereby releasing teachers for other more complex or esoteric asi)ccts of 
teaching writing. 

n. Hypothesis 

Using repeated tests at frec|uent intei vals to promt'te learning (Dempster, 
pp. L*ld J17), we have been able to see impro\ed exam scores and smaller standard de 
viations throughout the entire student population as the higher le\el students reach mas 
tery (juickh and the hnver level stiuients catch Uj). However, we were uitable to see how 
the students used their skills outside of class and thus could not determine if they wi-re 
incorporating the lessons from class into their use outside of class the real goal of 
teaching communication in hhighsh. The only way the students wen* consistently using 
the laiglish skills thev learnt in class (with the help of the repeated tests and otlu-r 
iiuxiiis) had been the journals which wen* written at all hours during and in between the 



assigned to the teachers at the school wherein this study took place* included discrete 
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school sessions. The journals, then, provided the only way to measure their actual use of 
the laiiKua>?e as they employed it outside of the classroom environment and was probably 
the closest thin^ possible to their unmonitored and unsupervised use of the language. 
Therefore, it is reasonable to hypothesize that if the students are using their skills 
learned in class outside of class, an analysis of their journals ma>' show a significant in- 
crease in the students’ correct use of the articles. If not, the analysis of their journals 
should show that the average scores of the definite and indefinite articles correctly used 
would not increase significantly whereas the scores of the short quizzes on articles 
given to the subjects inside class should increase as the term goes by. 

We can also predict that those students who are already at a fairly high level of 
competence in the use of articles would probably not show significant gains in their use 
of the articles, but neither should they regress, the students who are seriously deficient 
in their use of the articles and similar determiners should show a steady improvement in 
the e.xams, and may show improvement in their use of the determiners, and the survey 
should show that the standard deviations for the lower groups compared to the upper 
groups and the intia group scores of the lower level students evinced a smaller and 
smaller range as the total population reached parity in their use of the articles. 



N. Methods 

A. Research Design 

I'inding an efficient use of the teachers’ time in the course of a school year is a 
priority in large class situations. This is even more pronounced in writing classes that 
must employ a heavy production load that is to be monitored and evaluated by the 
teacher. There exists the technique of continuous testing that has shown significant 
gains in students’ test scores, but writing teachers (as do any other professional educa- 
tors) must also concern themselves with developing the students’ ability to communicate 
in the written medium of expression. In a study of Id'2 students in a post-secondary 
voeatirnial school in Tokyo Japan, a continuous testing regimen employing short quizzes 
and a short discussion of the results was employed exclusively to instruct the students 
in the use of definite and iiulcfinite articles and the students’ actual writing assignments 
were then survey(‘d to ascertain their proficiency in the use of articles and related de 
teriniiKMS. 

We chose the discrete objectives of definite* and indefinite articles and related de 
terminers as Ihe indepeaulent variables (Ary, Jacobs, Razavieh, 1990, p. dl) in the test 
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rclost inlcrvtMUion bocausi- llu*y arc parlicularly importanl for EFL in japan: They are 
more easily clcfinocl than other objeelives; they are arguably an important function of 
precise communication in Finglish; they have no corollary in the Japanese lanj^ua^e; and 
Japanese lend to neglect their use in speaking and writing. 

The number of limes the students used the articles and other corresponding de 
terminers correctly and incoi reelly in their writing assignments were tabulated. The stu- 
dents scores were then recorded and statistically analyzi'd. The criteria of evalualicm 
are as follows. 



Tal)U‘ 1 

Tabk' of all -<pic*ifu- (U tn niinrrs (llallidn> and Hasan. 1U7(>. 70) 



nomoiistraUVf 



H(jssi'ssivc 



Kofcifiitia! Sfk'cliW’ this, that 

these, those 

Non srU ot i\ V the 



lntc*rroj:aU\ o 



which 



Sporrh my. your, our 



OiluT his, her, their 
its 

one’s 



whose 



The students' randomly selected journal writing samples were evaluated with re 
gard to determiners that correspond to the position and function of the articles 'a\ 'an' 
and 'the'. Those that co-occur with nouns in the article position and do not act in the 
position of the head were considered in the students’ usage with liie following provisions 
(see table 1 ) : 

1. Adverbial (eircumstanlial) demonstratives, (here, there, now, then , etc., as defined by 
llalliday & Hasan, 197(J, pp. f)7 f)S), were not included: 

a) The textbook does not address the location of a process in space or time except to 
indicate if a noun is definite (particular) or indefinite (general) or has been mentioned 
before in the text (new or old information): 

b) riiey are typically usc*d as adjuncts in a clause and not as eUmients in the nominal 
group that precede the noun, they would not demonstrably provide the students a 
direct means of associating their similarities; 

2. The demonstrative refi'ienees were counted when used properly because they are 
easily used in place of the articles and also carry the meaning of plural and singular 
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nouns Uhis, that, these, those); 

‘3. Possessive determiners that act as modifiers, my, \our, our, his, her, their, its, one's 
{delineated by Halliday <S: Hasan, 1976, pp. 38, 45, 54) were initially counted because 
they occupy the article position (my part time job the part time job), but they were 
excluded from th(‘ stud\ because the students showed a clear mastery of their use: 

4. Non specific deictics (each, every, any, either, neither, no, some, all, and both) with the 
exception of the indefinite article a' were all initially counted except when they func- 
tioned as the head of an elliptical nominal (Halliday c^- Hasan, 1976, pp. 100- 101, 157- 
59), but they too were excluded when it became clear that both student groups evinced 
no significant difficulty in their use. 

5. The definite aiticle ma\ operate exophorically when there are examples of common 
use (I went to the hospital) and such use was counted as correct, otherwise the students 
were expected to establish nc v information with 'a' and then proceed to use the definite 
article (Halliday Hasan, 1976, p. 70). 



B. Procedure 

To check the reliability of our scoring procedure, a random sample of ten journal 
extracts from each group were photocopied and then marked. Three weeks later the 
same journal extracts were again photocopied and marked and then the second marking 
was compared to the first marking. These two tests were averaged together. The agree- 
ment between the two markings indicated an acceptable" tlegree of reliability in the eva- 
luation of the students’ journals. 



Tahir J 

Sforr h.ihilil> 



1st 



2nd 



niran 

s(lr\ 



!> I 



a.qi rriiu nt 
a^'irriiu'iU 



::u 

i;tsi 



7’j.ir^ 

la.o: 



The subjects in the c xpm imental group were in six different classes for a total of 
two hundred and forty. 'I'hese students were in their second semester of the 93-91 year 
class, taking an introductory level composition class at a post secondary vocational 
school that specialises in teaching fotangn languages for entry into the translation, inter- 
pretation, lour guide, and travel agency businesses. Only one hundred and thirty two 
students of the total group were measured because the other one hundred and eight did 
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not meet basic compliance guidelines. 

The subjects in the initial study ^roiip were in six classes of two- hundred and 
forty eight students in their first semester in the 93-9-1 class of the same institution 
with the same curriculum. Only two-hundred and twenty-two students of the total group 
were measured because the other twenty -six did not attend enough classes or did not 
write journals that would have given them the {)pportunity to use what they learned in 
class. 

The experimental subjects took a weekly series of short quizzes specifically cun 
structed to incorporate textbook objecti\cs in the use of dett*rminers (specifically arti- 
cles). whereas the students in the control group did not. The quizzes were fill-in the- 
blank that focused entirely on the determiners. gi\en at regular intervals in each class 
and the results were discussed with them for a total of ten to fifteen minutes per class 
(sec appendix). The quizzes were tabulated to ascertain whether the students were show 
ing any improvement in scores. The one-hundred and eight subjects not included in the 
study were excluded because they did not take at least 50% of the tests and or show 
evidence of ha\ ing done any writing which would have presented the opportunity to use 
what they had learned. 

In addition the subjects also had a series of letter writing assignments and main- 
tained journals throughout the semester. Since the groups in the study were assigned to 
their classes on the basis of their results on The Oxford Placement ICxam. they had. in 
effect, been stratified. For this reason, we ehose to sample n proportional number of stu- 
dents from each le\el to reflect the proportion in their classes (Ary. Jacobs. Kazavieh. 
1990: 173-75). The student’s journals in both groups were randomly sampled to mea- 
sure their use of the articles and other determiners. Twenty four students four from 
each class, were chosen to be examined in both the experimental and control groups. The 
use of the determiners was classfied as; 

I’snl c'otn‘Cll> l\<'(! incontrlly j InctMurlly oniKtrei 

yes no ^ () 

Data was evaluated for the correct us(* (y). incorrect use (n) or incorrect omission 
(o) of articles and related di'termiiU'r s \ 

It should be noted here, howawer. that this experiment could only (pialify as a 
pre experiment (Nunan. 1993. pp. -10 11; I. arsen Freeman. Long. 1991. pp. 19 31) for 
the following reasons. 
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a) There were pre- and post tests' in the form of the journal siirvevs taken at the begin- 
ning and the middle of the semester; 

b) There was no control group during the observation and the group surveyed initially 
did not qualify as a control group. Although it was composed of 1st semester students in 
the same class, 93 9 1. they were neither chosen randomly nor were they at a similar 
maturation level in their language studies. .Additionally, the students' journal composi 
tions were surveyed at different times in the course of the semester. Both the initial and 
e.xperimental groups' writing samples were taken at the beginning of the semester (prior 
to the testing regimen for the experimental group) and were analyzed for a baseline 
study. However the initial groups' writing was sampled again at the end of the 1st 
semester (after nearly o months in class) and the experimental groups writing (surveyed 
after the testing regimen) was sampled near the middle of the second semester - after 
nearly 8 months of instruction, 

c) There was no random assignnuuU of class placenu'iU as such. The students were 
placed in their respective classes on the basis of a perceived difference denoted by the 
‘Oxford Placement Test' (OPT) and the students themselves had selected their study ma 
jor which also determined the class in which they were placed. The students' course of 
study did not differ from the students in other courses with regard to thedr foreign Ian 
guage studies, merely with regard to their vocational studies. However, with the excep 
tion of the possible influence of the factors mentioned, all of the students in the mea- 
sured selection were assigiu'd to classes in this study in such a way that all the levels of 
competence according to the OPT were included in the initial and experimental groups; 
there was a reasonable cross section of the populations' level of ability. 



V. Results 

Table 3 and -1 summarizes the results of the experiments. Table 3 shows the re- 
sults of the scores tabulated from the journals twenty-four students wrote at the begin- 
ning and the end of the semester. Tabl(‘ 3 shows the mean scores of the journal survey 
and the percentage of the articles used correctly. A T t(*st was conducted to exainim* 
whether their increast's are statistically significant, I'able I gives the same information 
for the controlled group. 

There* was a 7,i\% incrt*ase in the* controlled study groups' samiih* nu*an (63 
7'3.(i‘Hi) and a 8.0% increase in the standard deviation. However, neither increase was 
statistically significant (t ■' 1.0-1 n. s.). The range of the* population mean was calculated 
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(after W'onnacott Wonnac'otl, 19cSn, p[). 228 229) and there was an increase in the 
ran.i^e (52 77% to 61 8 lV<)). By the same token, the experimental group evinced no 

significant increase in their sample mean (78 — 78.7%) although there was an increase 
by 1.6 % in the standard deviation (10.2 — ^ 1 1.8%). There was also a slight change in 
the population range (72.7 — ► 82.2'^(» to 72.5 ^ 85%). However, the result of the T test 
shows that the increase in the overall mean scores is statistically noii-significani for this 
group as well (p > .05). 

The I »wer ability groups' mean’ (n - 9) in the journal sur\ey declined somewhat 
(7 1.8 *-♦ 7T^o) and their standard deviation increased (9.7 — 17.15%). On the other 
hand, the higher groups' mean (n = 8) in the journal survey slightly increased (79.2 
81.7), but not statistical!>' significant enough (p = .59, n.s.). 

The average scores of the weekly short (piizzes given to the students in (he ex- 
perimental group increast'd by 10.1 (67.6 — ► 77.6) . The mean scores of each short 

(linz were (S7.6, (S8.8, 7 1,(S, 69.(5 76.0, 77.7, which showed a steady inereas(‘ as the 
semester went b\-. The drop betweeii the third and fourth mean scores corresponds to 
the student's vacation where in the score after \ acatioii was lower. 

To sum up the* i\*sults, neither experimental nor ceuit rolled grouj) showed any sig- 
nificant increase in the scores of the articles used correctly in journal writing. However, 
the scores of the short (luizzcs giv(*n to the exiHuiniental group have consistently in 
creased as they wc*re gi\en instructions and reviews on articles in el ass. 

V[. Discussion 

Possessive determiners (my, our, his, etc.) and selective demonstratives (this, that, 
etc.) have eomi)lements in Jai)anest- and apparently are not particularly difficult for the 
students to incorporate in tlu*ir writing in Bnglish. During the eoiirse of tabulating the 
results, the iK)ssessive determiners were eliminated from the analysis. The possessive 
determiners were used fre(|uc*ntly but the total population of both the initial study group 
and th(* exiK*rimental group niadi* a sum total of 12 mistakes in their ust* of the posses 
sives and 5 of those niistaki's were made by one student. Phe seh*ctive d(*terminers, on 
the other hand, were iis(*d infre(iu(*ntly and there wen* no mistak(*s for the entire popula 
tion. They were m*verthc*less count(*d in the tabulation since they could be used inter 
changeably by tin* students. 'Phe non speeifi(‘ d(*icties ((*ach, ev(*ry, any, (*te.) were also 
discounted since they were used incorrectly only eight times in the entire survey. 

The vast majority of mistakes were in the omission of the students' use of the 
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articles rather than their incorrect use. This would seem to support the view that they 
do not use them since they have no correlating words in Japanese. This failure to use the 
articles at all rather than using them incorrectly was a revelation. It indicates that there 
might be the need to approach the teaching (•; the articles through the use of the stu- 
di'uts’ texts rather than those prescribed in the textbook because the situations wherein 
they use the articles are not sufficientU' covered by the textbook. 

There were no significant changes in any of the measured parameters for either 
group. There was a small change in the number of misused articles but the incur n‘ct 
omissions were unchanged. Possible (‘xplanations for this lack of change, were not 
addressed by the slud\ . 

The students in the initial survey showed marginal changes - more so than the 
experimental group. These changes initially led us to believe that their progress could be 
accelerated. How(‘ver if the differences between the beginning point of the initial survey 
are compared with the end point of the experimental survey, the change is still insignifi- 
cant. 

The experinuuUal group showed ver\- little increase in the journal survey results 
(78% 78.7) despite the fact they were consistently given the reinforcement by the 

weekly short quizzes. The control group showed some impi'ovemcnt (6f).l 72.6%), but 

this improvenuTit is not a significant one. (p=1.0 1) The l esults of the quizzes they took , 
howevei', showed a steady ineiease {(^7.6 -^77.6%). These two facts seem to reveal an 
extremely interesting finding, i.e. they were not using what they learned in class in the 
journal w ritin. . a real out of class communication medium. They were learning determin 
ers in class for the in class use, mostl>‘ to improve the scores of the next quiz, not for 
the purpose of improving their communication skills in English. This suggests that what 
language learners can easily do in class may not be as t‘asy as when they face a real 
communication situation outside the classroom. 

One incident illustrates the above [)oint well. One of the authors took Japanese 
college stiulents to a TS college for a shoi t term stay. He noted that most of thi“ studi*nts, 
whos(‘ English was at the inteiniediate level, had trouble in a communication task as 
easy as introducing themselvi'S when they actually had to talk to theii' American room 
mate on the first day of their stay. 'PheN* completely forgot to say, " Hello." My name is 
so aiul so. Nice to meet you." When they were greeted with "How aie you doing?" from 
their roommate for the first time, they eouldirt possibK- imagine the expression wa^ a 
\ariation of "How are you.''", one of the (*asii'st and first extu cssions tluw' U*arned when 
tlu‘V started to study English at 12 years of age. Some of the participants on this prog 
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ram expressed the view that they had never thought their Engiish was as bad as it was. 
Thus, applying what they learned in a ‘safe’ classroom environment to an actual coni 
municalion is not as easy a task as language teachers may imagine. 

FVdagogicaily, the above fact also suggests that the language teachers may need to 
employ the communicative approach in class in a more focused manner. (living the lear* 
ners as many chances to use the language in a meaningful way as possible seems to be a 
key element in light of our findings. The teachers may also need to give the learners the 
chances to experience a variety of “Englishes" and a variety of situations. For example, 
they may need to have their students listen to English with some accent, wnth some 
noise, under some mental pressure, or beyond their students' comprehension level. Quiz- 
zes or r(*gular mid term or final exams may also need to be more proficiency oriented 
rather than achievement oriented ones if they believe the real purpose of teaching k'n- 
glish is for the learners to use the language when they need to in the outside world. 

I'he most important finding from the above mentioned facts is probably about us- 
ing test scores for measuring students’ improvements of their language competence. Our 
findings suggest that wt‘ may not be able to. or sometimes should not, use papcr-aiid-a 
pencil tests for measuring affects of any language teaching methods on the eompetence of 
the students. This is because theyv may do well in the tests trying to measure their irn 
provement of the target language items, but tluyv may not be able to apply them to out 
side class language use, which ta'searchers arc* supposedly interested in measuring. The 
language teachers and researchers may need to keep it in mind that what their students 
show in the test performance may he* comph'tely different from what we may call “com- 
municative compc'tence". In other words, scores of the tests givt'ii in language classroom 
may mean soim*thing only inside the classroom, not outside. 

1'he initial conjecture* that tin* lower level students would show the most change 
can not be suppi'rted in light of these* results but the overall level of writing may have 
chaiige*d. In neither case was the level of language investigated ■■ the students may have 
actually made ngnificaiit improvement in the language skills to a level whe*ie the use of 
d(*lerminers and articles are more* complex and therefore more* difficult. Hut the study 
would not have* shown this or taken it into account. 

riu* lower ability students' group nu*an in the journal survey dc*clined (74.8 
7 !.()%) and the standard deviation for the* lower level stud(*nts did not get smaller, 
which is opposite from what we had predicted. It in fact increased from 9.7 to I 7.f). 
This ma> mean that the lower level students had more difficult) utilizing what they had 
learnc*d in class in actual journal writing than upper level students. Some low level stu 
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dents may have got more confused with the use of articles as the instructions on articles 
became more complex towards the middle of the semester and could ''ot use what they 
have learned in the actual journal writing, whereas other low level students started to 
apply the information they have acquired in class to journal writing. To sum up, it 
seems the lower level students tend to fail to use the information learned in class in 
actual communication more than upper level students do. 

The high ability students' mean score in journal wTiting did not increase eitlier, 
which supports our prediction. This suggests the difficulty in acquiring accurate know- 
ledge in article use even foi' high ability students. This is most probably due to the na- 
ture of the Japanese language, which lacks the concept of articles, but this cannot be 
proved by this study. The fact that even high level students had difficulty in transfering 
language information to language skills means that it was not because of the level of pro- 
ficiency of the language that the subjects failed outside class to use what tliey learned in 
class. Therefore, it can be concluded that language learners in general would have diffi- 
culty in transfering in-class this specific language information to outside-class language 
use. 

The last thing we wanted to test was whether the in class continuous testing reg 
imen on articles impi’oved the accuracy and the fliumcy of tlu‘ articles used in the jour- 
nal writing. Judging from table 3 and -1, something we may infer is that tlu‘ short cjuizzes 
did not have significant effc'ct on the scores on the jcuirnal wiiting in the experimental 
group {7(S.O 7(S.7) (t = .67, n.s). On the other hand, surprisingly enough, the average 

scores of correctly used articles :or the journal writing in controlled group improved 
more drastically than that in experimental group (Gb.O 72.6). flowever, the increase 
is not yet statistically significant (t = l.O l, n.s). We do not know exactly why the control 
group without short weekly Cjuiz/.es have shown more' improvement. One possible' ex 
planation is that those who v ere in the controlled group could study articles in a more' 
relaxed maniu'r because they did not have to take* cpiizzc'S so often. I hey could have 
practiced article's in class with a long term olqective in mind, which is to use them in the 
journal writing, wdu'reas those' in the experimental group weirried teio much te) improve 
the (.juiz sce)res: thus, faile'el to use them ceirrc'ctly when they had tei use' tlu’in feir ceim 
numicatiein. V\'hen the Cjuizzes were over, they might have forgeitten all about what they 
have le'ariu'd in class. At any rate, it seems that the numerous weekly sheirt (piizze'S 
si'r\e*d e)nl\ to mak«' the' stiide'iits study feir the' eiuizzes ihemsedvcs, not to improve ceun 
munications skills using articles. 
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W. Shortcomings and suggestions 

Further study into these areas are certainly indicated. What is needed is a valid 
method of evaluating the students* overall writing skills and a thorough analysis of the 
precise way in which they use the articles and related determiners. 

Another difficulty was that during the time of the study, the teachers themselves 
carried student loads that usually exceeded 10 classes a week. Because the large class 
loads (40+ students) and inadequate time (80 minutes once a week) in class prevented 
the teachers from consistently dealing with the problems and deficiencies they saw first 
hand and made the goal of mastery learning problematic. For this reason, this study was 
undertaken in the hope that it would facilitate the development of a workable approach 
that would be manageable for the teachers within the restrictive parameters so common 
in foreign language education in post-secondary vocational colleges in Japan (Shiozawa, 
T., Simmons, T., & Noda, K., 1993: Shiozawa, T. & Simmons, T., 1994). 

The (|uasi-experimental design (Seliger and Shohamy, 1989, pp. 1 48- 149) offers 
a natural extension of this stufiy because there were contiad groups available at the in- 
stitution. .‘Ml students have had a standardized pre test that would have allowed com- 
parisons with other students in other classes that were not subjected to the independent 
variable. When the teaching schedule ended at the end of the semester, those groups 
could have been delineated and studied and the survey of the students in the ex- 
perimental group could also have been conducted again at the end of the semester. The 
intermediate evalualicin that this study represents could enable the investigator to use 
more accurate data in devidoping the tests themselves. For these reasons the study de 
sign can l)e usetl io accomniotlate a second evaluation. 

\1. Conclusion 

The stud(.‘ius' test score's impi'oved almost as expectc*d. 4'he parameters outlined 
by Dempster were easil\ duplicated with e)l)jecti\e criteria in this sur\’ey and the stu- 
dents behaved almost as predicted. lh)wever, even though the students at the IxUtom end 
of the class overcame the 73%+ score rangt‘ on their tests they did not show any im- 
provement in their use of the aiticles in jounial writing they were apparently not 
transferring their ktunvlcnlge to tludr skills. The students who scored w(*ll on th(» tests 
did not show much improvement but tlien it is difficult to slunv much relatise improve 
meat above 73%. So the test scores for the higher students could not really show signifi 
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cant iiiiprovemeiit. 

The hypothesis that the lower level students (below the 7b% range on the tests) 
would show improvement in skill can not bo born out because the lack of significant 
gains as indicted by the journal survey. The assumption that test scores indicate the im- 
provement of their skill in using- the’ articles was unfounded. This indicates the need to 
be wary of any similar assumptions in the classroom. This also points to the teachers’ 
limitations. In so far as educators are supported in their attempts develop the opportuni- 
ties to directly relate the educational objectives is the extent to which they can be 
actively involved in teaching the very skills targeted. Using tests, at least in this area of 
knowledge, does not adecjuately evaluate the students' knowledge and skill. 

As stated earlier, the use of continuous testing as a large class management tech- 
nique is effective in that it allt>ws the teacher to monitor and evaluate the class for admi- 
nistrative purposes. However, this approacli, which has been successfully duplicated 
numerous times, merely [)oints to the deficiencies in the t*ducational environment faced 
by students and teachers. In other words, e\eii though t*xam scores can bt* demonstrably 
improved, this does not mean the students have actually learned the targeted skill of us- 
ing the information: they are only improving their exam scores. 

It is obvious that it takes time foi* language learners to develop language informa 
tion into language skills, the competence they can actually use in communicatit)n. For ex 
ample, they may be able to pronounce a certain target pronuiiciati(Ui it(*m one day, but 
that does not guaiaiitee that they could do the same thing with the same ease the next 
day or that they could pronounce it correctly in an actual communication with native 
Knglish speakers. Our research highlighted this simple fact. If it servts to remind Ian 
guage teachers to be a little patient in strenuous teaching conditions ex|)erieiice(l every 
day and warn language researchers to be extra careful in interpreting test results given 
in class, this study, whieh took so much time and energy of oui' students and ouiselves, 
was worth conducting . 



■ Jat);m College of I'(jreign banguages (Nilioii Caikokugo Sennitm (iako) in Tokso. Jaj)au, a sul) 
sidiary of the lUiiisai (iakmui 1‘ouiidaiioii in Osaka. 

The process (>f rating intra marker reliability is similar to that usi-d in statistically evaluating 
reliability in r i’ading radiographs. Comments are made and one mouth later, without an refer 
enci‘ to the prevmu'-- marks and comments, tlu- i^rocess is (loiU‘ again and then the two are 
com}>ared. 

‘ Not nicorpoi ating coiii'et omissions into the measured parameters might ha\e lawealed some 
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inlcri’sting data but the logistical load of deinonstratinj? those measured parameters and then 
actually measurin^^ them is somothin}2: that could nrRuably onl\- bo done with such a lar^tc sam- 
ple through the aid of sophisticated software beyond the resources of this study. It woukh for 
example, included the counting of all nouns and the correct omission of the articles. 

' Students whose test grades were in the bottom 1 3 of the class, 

’ (Jiven the genre which was focused on the students personal experiences in what were pre- 
dominantly narratives, the frequent use of possossives is not surprising (K. Marriage, Aston 
I niversity, personal communication, 3 94) 
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Appendix 



A. textbooks 

Tiu‘ textbook iisi-d cluntiK the study incorjjorated a nuulest section on tlie use of deter 
miners which has proven inadecpiate foi‘ the students' use in tlunr own writiitK and in addres 
stUK the questions thi-y ask al/oul the use of tlie deternuners. 

A seh’ction was made from a wrilin^ specimen of the selected stmieiits and the proee< d 
iuK categories were ascertained aecorilin.ij to the following .qnidc“lines {Word for Word " . p. dli) 

1. Person, ('‘'untry. Lily. 

.Most names of tliinj^'s and peoj)le do not take an article 

2. Particular (definite), (ieneral (indefinite): 

Particular thin>ts often will take definite articles 

Countable, rncountable: 

rnconntable nouns when used in a j>(e!ieral sense do not have an article 
1. Singular. Plural: 

Singular. c()unlable nouns must have an article or a woid such as my, this, and so on. Plural 
nouns when use in a yteneral sense have no article. 
r>. New. Old: 

New information often has an indefinite article but tliinyts that have been meiitiom'd before often 
have a definite article 

B. Test Samples Used in the Testing Regimen (Taken Verbatim from the Text Used in 
the Classroom) 

1 . Word for Word I nit T 

Writing test for subjects covered on pa^e 22 d \ 

I 'se (1. an, the or 'X' if none i'^ nei'dcd. 

My famih went to .\ii/ona in 1 Pti 1 and we visited (Iraiid Canvoii. I had never 

seen Canyon. It was very beantifni. We stayed at hotel wlule we isited 

Canvon. Karly in morning. I would leave the hotel and ! would to bottom of 

Canyon on horseback. I always rode- same luuse. horse was very 

biK and gentle. It is very pleasant memory for me. 

'2. Word for \\ Ord Tnit T 

Writinj< test for subjects covered on pnm' 22 d 1 d.dM 
I'se a, an, the or if none is needed. 

'Peresa is friend of mine. She met star of "Dances with Wolves”. Kevin Costnei. 

She is student of .Amerindian cultures and wanted to ask him about l.ako 

t<i and Cheyenne pt‘ople vvlio wen* in movie. When she saw movie, she 

was surprised that it seemed so authentic. I’snally movies about .Amerindian |)eoj)les are 

v(‘rv strange and untrue. She says that Costner enjoyed making movie and has since 

made main .Amerindian friends. He (‘ven speaks litth’ Mandaii and Kiowa Ian 
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The folio lest was taken directly from the text book. The answers to the tost were 
^iveit at the back of the book. The test itself was ^>:iven at the end of the third month while 
the unit itself was tauijhl at the beKinninj^Mif the first nionlli in the semester. 



I nit 7 Woi’k with a partner to decide which article is lu^eded where you see in the 

following text. You do not need an article in every blank 

The l.ion 

Male lions are larmier than females and have thick niaiies on their necks and shouUlers. 
lions rest for up to twenty hours a day and live in nroup called pride. lio- 
ness d(.*es most of lumtin.u and is able to catch lar^re animals like antelo[)es and zebras, 

Lu)iis can have cub in any season. Between one and six youn>x can bv born at 

time, 

(i. W Ortl for Word I nit 7 

Writing test for subjects eoveretl on pa^v dd I P-2.2\\ 

I se (i, (DU the or ,\* if none is needed. 

students at Kofu West lli,uh School ha\e read book that is about 

animals that live in North .America. .After tlu> students read book, they saw 

iiu>\ie about herd of bison in Colorado Mountains. bison herd is 

•Nafe tioin luinlers and there are more bison each year, hunters used to hunt them 

without any control and herds of bis(»n he^an to disap[)car. Now hunters 

can not hunt unless they ask permission and pay money. 

( eon eet aiiswei s (ca) 1 7 ) 
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